Multi-Gate Attention Network for Image Captioning

Self-attention mechanism, which has been successfully applied to current encoder-decoder framework of image captioning, is used enhance the feature representation in encoder and capture most relevant information for language decoder. However, existing methods will assign attention weights all candidate vectors, implicitly hypothesizes that vectors are relevant. Moreover, self-attention mechanisms ignore intra-object distribution, only consider inter-object relationships. In this paper, we propose a Multi-Gate Attention (MGA) block, expands traditional by equipping with additional Weight Gate (AWG) module Self-Gated (SG) module. The former constrains be assigned contributive objects. latter adopted distribution eliminate irrelevant object vector. Furthermore, captioning apply original transformer designed natural processing task, refine features directly. Therefore, pre-layernorm simplify architecture make it more efficient enhancement. By integrating MGA block into AWG decoder, present novel Network (MGAN). experiments on MS COCO dataset indicate MGAN outperforms state-of-the-art, further other combined blocks demonstrate generalizability our proposal.